texts:
# all imports
from IPython.display import Javascript
from google.colab import output
from base64 import b64decode

RECORD = """
const sleep  = time => new Promise(resolve => setTimeout(resolve, time))
const b2text = blob => new Promise(resolve => {
  const reader = new FileReader()
  reader.onloadend = e => resolve(e.srcElement.result)
  reader.readAsDataURL(blob)
})
var record = time => new Promise(async resolve => {
  stream = await navigator.mediaDevices.getUserMedia({ audio: true })
  recorder = new MediaRecorder(stream)
  chunks = []
  recorder.ondataavailable = e => chunks.push(e.data)
  recorder.start()
  await sleep(time)
  recorder.onstop = async ()=>{
    blob = new Blob(chunks)
    text = await b2text(blob)
    resolve(text)
  }
  recorder.stop()
})
"""

def record(sec=5):
  display(Javascript(RECORD))
  s = output.eval_js('record(%d)' % (sec*1000))
  b = b64decode(s.split(',')[1])
  with open('audio.wav','wb') as f:
    f.write(b)
  return 'audio.wav'  # or webm ?

The above is the code for audio recording in google colab. But the audio file is not saved in google colab and showing after running the code. kindly help?


texts: One Year Ahead Forecasting with Unevenly Spaced Time Series

texts:
I have many products in my warehouses which can be "demanded" any day by my different clients. I want to forecast how many of each item will be demanded for the whole next year. Naturally, as I'm having a large amount of different time series (each one for a different product, with potentially some shared patterns and correlations between them), I want to train some kind of RNN (could be a simple LSTM or even something more complex like Amazons DeepAR or Googles TemporalFusionTransformer). Also, as the data is unevenly spaced (there can be days or even months without any demand), I'm planning to provide the model with a timestamp delta (dts) as an additional input feature.
The problem is, how can I use such trained network to predict the total demand for one year ahead? As the network trained with unevenly spaced data, I cant determine the time window for each new prediction, let alone a full year. How can I tweak the network to achieve this? Or maybe I am tackling the problem from a wrong direction? (I did try a couple of different solutions like Prophet who have been fruitless)
Thanks in advance.


texts: How can I add custom numerical features for training to BERT fine tuning?

texts:
I have currently fine tuned the BERT model on some custom data and I want to conduct some more experiments to increase the accuracy.

My original dataset consists of a pair of sentences (like MRPC dataset). I want to increase the accuracy of classification by adding some numerical features (which I will separately calculate). I wanted to know if I could train bert using this after I have already fine tuned it ?

I have read about some solutions people have proposed in the past like :
'Extracting word embeddings (extract_features.py on Bert GITHUB) and combining that with my custom data to feed a single layer CNN network.' I dont want to lose out on the accuracy the BERT network is providing me by only extracting features from the pretrained model.

So is there any way I can create a kind of hybrid model which first fine tunes BERT and then I add my features and it feeds into another model for an improved classification ?

P.S

As I am new to tensorflow and Deep learning, please let me know if there is something fundamentally wrong in my understanding. Thank you for your help


texts: How can I use geo location for predictive modelling

texts: While determining the house price I don't want to throw away geo location as feature. At least I want to see if there is correlation. Geo-location is more like categorical variable. Is there a way to make it useful feature


texts: What is the difference between BatchNorm and Adaptive BatchNorm (AdaBN)?

texts:
I understand that BatchNorm (Batch Normalization) centers to (mean, std) = (0, 1) and potentially scales (with $ \gamma $) and offsets (with $ \beta $) the data which is input to the layer. BatchNorm follows this formula:

(retrieved from arxiv-id 1502.03167)

However, when it comes to 'adaptive BatchNorm', I don't understand what the difference is. What is adaptive BatchNorm doing differently? It is described as follows:

(retrieved from arxiv-id 1603.04779)


texts: Is there a way to find the weights of every feature in spark ml model?

texts:
I am using spark ml classifier to create a model to predict whether a user will buy a product or not and I am using features both continuous and categorical. The model is predicting the outcome with some accuracy. Is there a way to see how much each feature is contributing to the prediction (i.e) the weight of each feature in the model created from training data? Because I need to know why a user will buy a product or not. I am using spark 2.0.0 and all the classifiers available in it

Thanks in Advance


texts: Examples on kohonen self organizing maps

texts: Is there a simple example to start with for using kohonen 1.1.2 or is it only the test file that will be the reference?


texts: Find suitable locations using Machine Learning

texts:
Just for fun, I am currently trying to find suitable locations to deploy new stores. So what I did so far is to take the actual sites of current stores and to assign surrounding variables to it. These features include for example: point of interest density, population density, region popularity etc. In total I have 9000, 100 dimensional points. 1000 of these points contain stores already, the remaining 8000 do not.

In the next step I want to perform dim reduction using PCA. However, I am not sure how to proceed afterwards. Should I try to cluster the points? Or how can I „predict“ which of the points are suitable candidates for new stores? Maybe using some kind of skip gram model?

Hoping to get some advise:)

Cheers,
Tom


texts: How to approach Customer dimensions and work hours relation problem

texts:
I am thinking about how to solve the following problem related to a supermarket and its employees' workload prediction:
I have customer dimensions (attributes):

CustomerID
Loyalty status
Uses e-store
Monthly Spend
Monthly Spend in e-store
Delivery %
Pick-up %
In-store purchase %

Interested in FTE requirements in the following:

Selves restocking
Cashier
Product compiler for e-store
Take-away

Based on the customer's behaviour, I would need to build a model to estimate the workload that the store must do. E.g. if the customer uses e-store, this means the store must have an employee, who compiles the products and ships them. When a person comes to the store, then a cashier is needed and shelves must be filled. et cetera. Each FTE calculation is according to a ratio.
My idea:
Cluster the customers according to their dimensions. Either K-means and using the elbow or by PCA to understand relevant dimensions.
Then build linear regression on top of it for each of the contributions to the FTE number. Sum the FTEs for the total effect.
Any other suggestions?
Much appreciated!


texts:
I need to convert a datetime.date(2007, 4, 30) to a Unix timestamp in pandas.

thank you.

my datable:

0     2007-04-30
1     2007-05-31
2     2007-06-30
3     2007-07-31
4     2007-08-31
Name: datetime, dtype: object


texts: Perform classification on market basket analysis

texts:
I have the following problem that I don't know how to solve:

I have the data for different market baskets with a corresponding class. So for example I know:

Student - {beer, milk, water}
Professional -  {nuts, pizza, bananas}
Student - {oranges, tomatoes, beer}
...

Is there a method to create a classification model so that I can use the content of the market basket in order to determine the corresponding class (Student, Professional, ...)? Thank you!


texts: Classification when variables are in ranges

texts:
I want to classify my data and some of my variables are ranges.

I classify location so for example, school, the hours that people are at school are from 7:00 to 14:00, some of my variables are categorical (working day) and some of them numerics(frequency of visiting in a month).

I thought using LDA but how can I declare a range as one variable?

btw, I use Python

Example:

place         visiting time    frequency    workday
school         8:00-14:00        18-20           1
restaurant    13:00-21:00        0-3            both
bank          8:00-17:00         0-4             1
night club     21:00-2:00        0-4            both

The algorithm should be supervised because I insert the ground truth (the place) by myself.

TIA


texts: why this naming convention for padding as "Same" and "Valid" in keras

texts: I was going through CNN's and found that padding argument should be set to "Valid" if i need no padding and "Same" if i need padding. But, it doesn't make any sense to me. Why can't keras development team just put "Padding = True", is there a specific reason to choose this convention?


texts: Which supervised machine learning algorithms assume normally distributed feature variables?

texts:
I want to understand the assumptions made by supervised machine learning models.

I've heard it said many times that 'you need to make sure your feature variables are normally distributed for your ML models to work.' However, when I looked up the assumptions for Linear Regression, I found many conflicting viewpoints.

This post and this post mention normality of error distributions. The latter even says that features do not need to be normally distributed - just the errors.

Statistics solutions says that features do need multivariate normality, as do a lot of top beginner ML courses such as Machine Learning A-Z.

Wikipedia says that features do not need to be normally distributed, but it has a major influence on the precision of estimates.

Which algorithms assume that the feature variables are normally distributed? For ones that don't, why is it beneficial to scale your features so that they are more normally distributed?


texts:
I have researched and am familiar with a few optimization problems. However, I can't seem to find material on what I'm attempting to solve in python.

Suppose you have 8 items of varying weight.

1;128
2;0
3;130
4;186
5;0
6;0
7;12
8;12

Let's say I have an unlimited amount of 'knapsacks'. I can only use 2 different items to fill each knapsack which can only hold a value of 90. Each time I load the knapsacks, I would like to then come back and make another pass with the remaining values.

For example:
On the first pass, I can fit item 1 into a knapsack, item 3 in a knapsack, and item 4 in a knapsack.

On the next pass, even though I took 90 away from item 4, item 4 still holds a value of over 90, and I can put item 4 in another knapsack.

Is there is a specific term for this particular problem or a common solution? I would greatly appreciate any direction.


texts: How to perform feature selection with Categorical Variables and Continuous Target, provided that data is not normally distributed?

texts: Basically I am trying a use Multi Linear Regression Model to predict the salaries of employees. I have a total of 88 dependent feature from which 19 are categorical and the rest are continuous. I have managed to reduced the number of continuous features from 69 to 41. Now I am trying to reduce the number of categorical feature, but since my data is not normally distributed I can't use t-test or ANOVA. Which other tests can I use to see if the features are significant to predict the target?


texts:
I would like to find different patterns recognition algorithm to detect different type of fraud. I have 1 million unstructured text documents about the clients' information with metadata about the client name, viewers, location in the cloud.
Here are the patterns I was thinking of and the related fraud that i would like to detect:

Numeric Patterns - fictitious invoice numbers, fictitiously-generated transaction amounts.
Time Patterns - transactions occurring too regularly, activity at unusual times or dates.
Name Patterns - similar and alerted name and addresses.
Geographic Patterns - Proximity relationships between apparently unrelated entities.

What technique can I use? any keywords?


texts: How to match a user with another user based on their taste?

texts:
Information available

Consider that there are N users on a platform. Every user adds items that they like on their profile. These items have static attributes that describe the product.

User A:
Row   | Attribute a | Attribute b | Attribute c
Item 1|    0.593    |    0.7852   |   0.484
Item 2|    0.18     |    0.96     |   0.05
Item 3|    0.423    |    0.886    |   0.156

User B:
Row   | Attribute a | Attribute b | Attribute c
Item 7|    0.228    |    0.148    |   0.658
Item 8|    0.785    |    0.33     |   0.887
Item 9|    0.569    |    0.994    |   0.374

User A has a list of items that he/she likes. Same goes with User B... User N. The items in the profiles of different users might or might not be the same but the items describe the User's taste for that particular item.

Goal

What I want to do is, match a User with another User if they have a similar taste in picking items. I don't understand how to achieve this. Any help is appreciated!


texts: How to train a Word2vec model so that particular words set would reside close to each other in the vector space?

texts:
I am supposed to build a resume parser. For the skill extraction part,  currently I am matching bi-grams and uni-grams in a CV against a predefined skill set, which is not that successful. Can I train a Word2vec model so that words and phrases in a cv which could classify as skills to have similar word vectors?

Is there any method which could be more successful?
Please help. Thank you in advance.


texts: I have a saved canvas I created in Orange 2.7. I just installed Orange 3 and opening the old canvas gives multiple "UnknownWidgetDefinition" errors. I gather that the naming/location of some widgets has changed between versions. Is there a way to import the old canvas?


texts: Roadmap to learn CNN in tensorflow from scratch

texts:
I'm working in the medical field and I'd like to learn applications of CNN for image recognition and classification. All the (few) things I learned come from self-learning on the web or sparse books. I'm studying now Tensorflow for CNN implementation but I'm having trouble finding clear documentation for my actual level, so I think I'm missing the basic knowledge in order to understand this.

I'm at a basic level of python programming, I have better understanding of classical machine learning algorithms, which resources should I learn in order to get a good grasp of the argument? Is there such an ideal pathway to this?


texts:
I have a price data set where on some days there are up to five data points and some days none at all. For example:

2.110   2017-04-19
1.910   2017-04-23
1.980   2017-04-24
1.980   2017-04-24
1.980   2017-04-24
1.230   2017-04-24
2.100   2017-05-04
1.920   2017-05-08

The total data set is around 500 points.

What is the best way to go about interpolating this data and making it's frequency daily?

I'm considering using Gaussian Process Modelling using PyMC3 but would really appreciate any ideas or advice. Additionally, the volatility of the price data must be preserved as best as possible as this will be input to a volatility model later on.


texts: Sharing industry data with academic class contracts?

texts:
Could someone point me to a standard data sharing agreement? I have an interesting data set that I want to share with my local university for  their Data Mining class. The data set includes time-series and email statistics. All domains and IPs are anonymized before release.

Main points:

Use at your own risk
Protect my company and client's privacy
Cite my company
Do not re-share without permission

Am I missing anything important here?


texts: Reliable way to verify Pyspark data frame column type

texts:
If I read data from a CSV, all the columns will be of "String" type by default. Generally, I inspect the data using the following functions which gives an overview of the data and its types

df.dtypes
df.show()
df.printSchema()
df.distinct().count()
df.describe().show()

But, if there is a column that I believe is of a particular type e.g. Double, I cannot be sure if all the values are double if I don't have business knowledge and because

I cannot see all the values (millions of unique values)
If I explicitly cast it to double type, spark quietly converts the type without throwing any exception and the values which are not double are converted to "null" - for example

Code:

from pyspark.sql.types import DoubleType

changedTypedf = df_original.withColumn('label', df_control_trip['id'].cast(DoubleType()))

What could be the best way to confirm the type of column then?


texts: A single column has many values per row, separated by a comma. How to create an individual column for each of these?

texts:
As you can see below, I have a column called code with multiple values per row, separated by a comma. How can I create a column for each of these codes and make them all binary values?

i.e. code6254, code5854 etc...., where all these columns will be of binary value 0 or 1 depending on whether that row has the code or not? Thanks in advance :)


texts: Do we need to do multiple times deep learning and average ROC?

texts:
Do we need to do multiple times deep learning and average ROC(AUC)?
Since we've might get different ROC(AUC) every round we train and test (by KERAS)
Is it necessary to average ROC(AUC) with multiple times training and testing?
(or just choose the best round?)


texts: Intuition behind the fact that SVM uses only measure of similarity between examples for classification

texts:
I have read about SVM and although I did not understand the math behind it completly, I know that it produces decision plane with maximum margin between examples of different classes and role of support vectors in the process.
I also know that SVM is a kind of dual learing algorithm(algorithms that operate only using the dot product between examples). It uses kernel functions to calculate dot product(measure of similarity) between training examples.

What I want to understand in simple terms is that: Suppose I have a similarity matrix of all training examples specifying amount of similary between any(all) two examples in training sample. How Can I make a classifier or cluster based only on this information?


texts: Transfer Learning on Resnets/VGGs -- Validation accuracy can never be over 75%

texts:
I am trying to classify skin cancer images into two categories -- malignant and benign. Literatures suggest that using pre-trained resnet/vgg network achieves more than 90% accuracy. However, with my dataset, whatever I try, the validation accuracy can never be greater than 75%.
I am using a well-balanced dataset where there are 500 malignant and 500 benign images in the training dataset. The number of images is on a smaller end; however, the fact that I am using transfer learning and other literatures showing that with only ~1000 images they could achieve more than 90% accuracy, I believe there's something wrong with the transfer learning logic than the data itself.
I additionally have tried the followings:

Used different number of batches (8/16/32/64/128) and different learning rates (from 1e-7 to 0.1)
"unfreezed (change require_grad=False to True)" the pretrained resnet/vgg after the validation error platues and updated the weights for smaller epochs (a.k.a. fine tuning the pretrained network after setting require_grad=True after validation accuracy platues)
Decayed the learning rate using scheduler
used other nets: inception v3, GoogleNet
rather than having one fully connected layer, used dropout-FC-relu-FC

This is the core logic:
model_ft = models.resnet18(pretrained=True) # some other place in my code, I set all of resnet18 param to require_grad=False
num_ftrs = model_ft.fc.in_features
model_ft.fc = nn.Linear(num_ftrs, 2)

model_ft = model_ft.to(device)

criterion = nn.CrossEntropyLoss()

optimizer_ft = optim.SGD(model_ft.parameters(), lr=0.001, momentum=0.9)

and this is how I normalize my data (I have tried cropping, rotating, and horizontal-flipping; none of these boosted the performance. For now, in order to preserve some relevant clinical features, I decided not to touch the data):
data_transforms = {
    'train': transforms.Compose([
        transforms.Resize(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'val': transforms.Compose([
        transforms.Resize(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
}

Can someone please let me know what I could do to improve the model performance? Without too much of a data pre-processing, we think the model should achieve well-above 85% accuracy in the validation set.


texts: Machine learning Classification model for binary input and output data

texts:
I have a large longitudinal dataset with 5 minutes granularity for a period of around 30 months from thousands of households. I want to classify them as binary output (0/1) based on the input which is also a set of binary variables (sensors activated or not- 0/1. I have a training dataset available with the labeled binary output(0/1) with binary inputs.

I want to know which Machine learning model will be best for this type of case where both input and outputs are binary in nature.

Whether Logistic regression is one of the options or not?


texts: Difference between cluster centers and means

texts:
The following is the output of the cluster centers I got from a cluster model (kmeans - 6 clusters)

  3.371069, 3.920354, 3.629747, 3.700000, 3.988506, 3.740385

However, after segmenting the data into the 6 clusters and taking the average of the data for each of the 6 clusters, I get a different set of numbers than the above.

  3.7, 3.6, 3.6, 3.8, 4.2, 3.3

My question is are cluster centers different than means?

Why am I getting different values?


texts: Accuracy decrease in production after adding additional input datas

texts:
I am trying to predict TimeSeriesA by using a CNN. I create snapshot images of the timeseries and these are then labelled.

With a very simple snapshot I get reasonable training and test accuracy. When I apply the model to real world in production I also get reasonable accuracy.

Inorder to improve the accuracy, I added other timeseries to the snapshots that may or may not add value.

Both my training and testing accuracy increased (Training much more so). However, my production accuracy has gone down greatly.

Why might this happen? The original data is still in the snapshot in exactly the same format. Can a CNN be confused (wrong word!) by the additional data?

Here is a look at the stationarity of TimeSeriesA:

Year: 2000 Ave: -0.0003 Std: 0.0076 Skew: 0.2166
Year: 2001 Ave: -0.0002 Std: 0.0072 Skew: 0.0158
Year: 2002 Ave: 0.0006 Std: 0.0056 Skew: -0.2445
Year: 2003 Ave: 0.0007 Std: 0.0065 Skew: -0.0402
Year: 2004 Ave: 0.0003 Std: 0.0067 Skew: -0.2640
Year: 2005 Ave: -0.0005 Std: 0.0056 Skew: 0.2420
Year: 2006 Ave: 0.0004 Std: 0.0047 Skew: 0.2711
Year: 2007 Ave: 0.0004 Std: 0.0039 Skew: -0.3177
Year: 2008 Ave: -0.0001 Std: 0.0087 Skew: 0.3768
Year: 2009 Ave: 0.0001 Std: 0.0076 Skew: 0.2327
Year: 2010 Ave: -0.0002 Std: 0.0074 Skew: 0.0112
Year: 2011 Ave: -0.0001 Std: 0.0074 Skew: -0.2599
Year: 2012 Ave: 0.0001 Std: 0.0051 Skew: 0.2541
Year: 2013 Ave: 0.0002 Std: 0.0046 Skew: 0.0818
Year: 2014 Ave: -0.0005 Std: 0.0039 Skew: -0.1489
Year: 2015 Ave: -0.0004 Std: 0.0076 Skew: 0.2973
Year: 2016 Ave: -0.0001 Std: 0.0051 Skew: 0.0076
Year: 2017 Ave: 0.0005 Std: 0.0045 Skew: 0.3101
Year: 2018 Ave: -0.0002 Std: 0.0045 Skew: -0.1658
Year: 2019 Ave: -0.0002 Std: 0.0033 Skew: -0.1124

I train the model with data up to end of 2010. I get a training accuracy and validation accuracy of around 65%.

When I then appy the model to data from 2011 to 2019, I get a drop-off in accuracy. The green years are those included in the training.


texts: Error while using lightGBM's cv() function for a regression problem

texts:
I am trying to use lightGBM's cv() function for tuning my model for a regression problem. My main model is lightgbm.LGBMRegressor(). However, I am encountering the errors which is a bit confusing given that I am in a regression mode and NOT classification mode. Below are the code snippet and part of the trace.

param_grid = {
    'class_weight': [None, 'balanced'],
    'num_leaves': list(range(30, 150)),
    'learning_rate': list(np.logspace(np.log(0.005), np.log(0.2), base = np.exp(1), num = 1000)),
    'subsample_for_bin': list(range(20000, 300000, 20000)),
    'min_child_samples': list(range(20, 500, 5)),
    'reg_alpha': list(np.linspace(0, 1)),
    'reg_lambda': list(np.linspace(0, 1)),
    'colsample_bytree': list(np.linspace(0.6, 1, 10)),
    'objective': 'regression'
 }
lgb_train = lgb.Dataset(X_train,y_train)
cv_results = lgb.cv(params, lgb_train, num_boost_round = 10000, nfold = 10, metrics = 'rmse', shuffle=False,
           early_stopping_rounds = 100, verbose_eval = False, seed = 50 )

Part of the trace of the error I am getting is

---------------------------------------------------------------------------
LightGBMError                             Traceback (most recent call last)
<ipython-input-84-3f76f5345fda> in <module>
      1 # Perform cross validation with 10 folds
      2 cv_results = lgb.cv(params, lgb_train, num_boost_round = 10000, nfold = 10, metrics = 'rmse', shuffle=False,
----> 3            early_stopping_rounds = 100, verbose_eval = False, seed = 50, )

/anaconda3/envs/py36/lib/python3.6/site-packages/lightgbm/engine.py in cv(params, train_set, num_boost_round, folds, nfold, stratified, shuffle, metrics, fobj, feval, init_model, feature_name, categorical_feature, early_stopping_rounds, fpreproc, verbose_eval, show_stdv, seed, callbacks)
    456     cvfolds = _make_n_folds(train_set, folds=folds, nfold=nfold,
    457                             params=params, seed=seed, fpreproc=fpreproc,
--> 458                             stratified=stratified, shuffle=shuffle)
    459
    460     # setup callbacks

/anaconda3/envs/py36/lib/python3.6/site-packages/lightgbm/engine.py in _make_n_folds(full_data, folds, nfold, params, seed, fpreproc, stratified, shuffle)
    315         else:
    316             tparam = params
--> 317         cvbooster = Booster(tparam, train_set)
    318         cvbooster.add_valid(valid_set, 'valid')
    319         ret.append(cvbooster)

/anaconda3/envs/py36/lib/python3.6/site-packages/lightgbm/basic.py in __init__(self, params, train_set, model_file, silent)
   1552                 train_set.construct().handle,
   1553                 c_str(params_str),
-> 1554                 ctypes.byref(self.handle)))
   1555             # save reference to data
   1556             self.train_set = train_set

/anaconda3/envs/py36/lib/python3.6/site-packages/lightgbm/basic.py in _safe_call(ret)
     44     """
     45     if ret != 0:
---> 46         raise LightGBMError(decode_string(_LIB.LGBM_GetLastError()))
     47
     48

LightGBMError: Unknown objective type name: r

I don't understand the error. Moreover, if I remove the 'objective':'regressor', then we are getting the error: y variable is only 1 class which seems to me to be referring to a classifier.

Any help would be great.

Thanks


texts: Ranked tag recommendation for university courses

texts:
Our system allows an admin to manage a database of university courses. These courses have multiple fields, like the department, a title, and a description.

I am adding the ability to add learning objectives to a course. To simplify the problem, let's say that learning objectives are just tags. Courses can have more than one learning objective associated with them. So a course like CHEM 101 might have "chemistry", "technology", "science", and several others.

Assuming I can reduce a course to a set of features, (using keywords/stemming/nlp, I suppose?), what kind of problem is this and what algorithm would you suggest? It seems very similar to a classification problem, but I want to provide a sorted list of suggestions with the most relevant at the top.


texts:
I am trying to visualize retweet network in order to find out which users are most likely to have most influence on other users. Here is my code:

    import networkx as nx

    G_retweet = nx.from_pandas_edgelist(translated_iranian_tweets,
                                        source = "userid",
                                        target = "retweet_userid",
                                        create_using = nx.DiGraph())

    print('There are {} Nodes inside Retweet Network'.format(len(G_retweet.nodes())))
    print('There are {} Edges inside Retweet Network'.format(len(G_retweet.edges())))

import matplotlib.pyplot as plt

#Size varies by the number of edges the node has (its degree)
sizes = [x[1] for x in G_retweet.degree()]

    nx.draw_networkx(G_retweet,
                     pos = nx.circular_layout(G_retweet),
                     with_labels = False,
                     node_size = sizes,
                     width = 0.1,
                     alpha = 0.7,
                     arrowsize = 2,
                     linewidths = 0)

    plt.axis('off')
    plt.show()

There are 18631 nodes and 35008 edges inside this network. Visualization is horrible, you cannot see anything. Does anyone have any suggestions what should I do about it? Should I try to extract specific type of users with specific tweets in order to reduce size of my dataset and then try to visualize the network, or something else?


texts: Predicting similarity between nouns like university names and tech companies?

texts:
I am trying to extract entities like university studied at and tech companies from resumes , I have a list of popular universities and companies and I want to find out which university best matches the extracted entity

Example

1) University in the list : IIT Bombay

Extracted entity : Education : Indian Institute of technology Bombay Btech

2)University in the list : Infosys

Extracted entity : Infosys India Ltd.

As you can see , there are extra unwanted words , short forms , expanded forms etc recognized in the extracted entity , is there any sentence similarity algorithms best suited for this purpose ?

Using SpaCy for entity extraction.


texts: How are 1x1 convolutions the same as a fully connected layer?

texts:
I've recently read Yan LeCuns comment on 1x1 convolutions:

  In Convolutional Nets, there is no such thing as "fully-connected layers". There are only convolution layers with 1x1 convolution kernels and a full connection table.

  It's a too-rarely-understood fact that ConvNets don't need to have a fixed-size input. You can train them on inputs that happen to produce a single output vector (with no spatial extent), and then apply them to larger images. Instead of a single output vector, you then get a spatial map of output vectors. Each vector sees input windows at different locations on the input.
  In that scenario, the "fully connected layers" really act as 1x1 convolutions.

I would like to see a simple example for this.

Example

Assume you have a fully connected network. It has only an input layer and an
output layer. The input layer has 3 nodes, the output layer has 2 nodes. This
network has $3 \cdot 2 = 6$ parameters. To make it even more concrete, lets say you have a ReLU activation function in the output layer and the weight matrix

$$
\begin{align}
W &=
\begin{pmatrix}
 0 & 1 & 1\\
 2 & 3 & 5\\
\end{pmatrix} \in \mathbb{R}^{2 \times 3}\\
b &= \begin{pmatrix}8\\ 13\end{pmatrix} \in \mathbb{R}^2
\end{align}
$$

So the network is $f(x) = ReLU(W \cdot x + b)$ with $x \in \mathbb{R}^3$.

How would the convolutional layer have to look like to be the same? What does LeCun mean with "full connection table"?

I guess to get an equivalent CNN it would have to have exactly the same number of parameters. The MLP from above has $2 \cdot 3 + 2 = 8$ parameters.


texts: I have to detect anomalies from my data-set. The anomaly is about in which area and in which time of network usage(total_activity in my data) drastically improved. Help me to know how to apply k-means for this data-set.


texts: How do I load FastText pretrained model with Gensim?

texts:
I tried to load fastText pretrained model from here Fasttext model. I am using wiki.simple.en

from gensim.models.keyedvectors import KeyedVectors

word_vectors = KeyedVectors.load_word2vec_format('wiki.simple.bin', binary=True)

But, it shows the following errors

Traceback (most recent call last):
  File "nltk_check.py", line 28, in <module>
    word_vectors = KeyedVectors.load_word2vec_format('wiki.simple.bin', binary=True)
  File "P:\major_project\venv\lib\sitepackages\gensim\models\keyedvectors.py",line 206, in load_word2vec_format
     header = utils.to_unicode(fin.readline(), encoding=encoding)
  File "P:\major_project\venv\lib\site-packages\gensim\utils.py", line 235, in any2unicode
    return unicode(text, encoding, errors=errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xba in position 0: invalid start byte

Question 1 How do I load fasttext model with Gensim?

Question 2 Also, after loading the model, I want to find the similarity between two words

 model.find_similarity('teacher', 'teaches')
 # Something like this
 Output : 0.99

How do I do this?


texts:
I am trying to build a regression tree with 70 attributes where the business team wants to fix the first two levels namely country and product type.To achieve this,I have two proposals:

1.Build a separate tree for each combination of country & product type and use subsets of the data accordingly and pass on to respective tree for prediction.Saw here in comments.I have 88 levels in country and 3 levels in product type so it will generate 264 trees.

2.Build a basic tree with two variables namely country and product type with appropriate cp value to generate all combination as leaf nodes(264).Build a second tree with rest all variables and stack tree one upon tree two as a single decision tree.

I don't think the first one is the right way to do.Also, struck on how to stack the trees in second approach, even if it is not the right way would love to know how to achieve this.

Please guide me to approach the problem.Thanks.


texts: How can I make my own Neural Network model for Object Detection?

texts:
I'm using the ImageAI module in Python3 to do some object detection on some images I scraped from a video game. In testing, I am able to successfully detect normal world objects from a test photo of a bus and some people using a RetinaNet model that is trained for those normal world objects.

However, my objects are not normal world objects such as buses, cars, people, etc. They are 2D models in a video game. Thus, I guess I need to try and build a new RetinaNet type model to load into object detection class.

from imageai.Detection import ObjectDetection
import os

execution_path = os.getcwd()

detector = ObjectDetection()
detector.setModelTypeAsRetinaNet()
detector.setModelPath( os.path.join(execution_path , "resnet50_coco_best_v2.0.1.h5"))
detector.loadModel()

How can I make my own model (.h5) trained to the objects I really want to detect? I hope this is not too broad. Majority of the information on object detection are using pre-defined models that are already trained to detect pretty common objects. They really don't go into depth of how you train your own models to use against objects that maybe are not ideally classified by these models.


texts: How to use text classification where the training source are txt files in categorized folders?

texts:
I have 200 *.txt unique files for each folder:

Each file is a lawsuit initial text separated by legal areas (folders) of public advocacy.

I would like to create a training data to predict new lawsuit by its legal area.

Last year, I have tried use PHP-ML, but it consume too much memory, so I would like to migrate to Python.

I started the code, loading each text file in a "json-alike" structure, but I don't know the next steps:

import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer,TfidfVectorizer
from sklearn.base import TransformerMixin
from sklearn.pipeline import Pipeline
import os

path = 'C:\wamp64\www\machine_learning\webroot\iniciais\\'

files = {}

for directory in os.listdir(path):
    if os.path.isdir(path+directory):
        files[directory] = [];
        full_path = path+directory
        for filename in os.listdir(full_path):
                full_filename = path+directory+"\\"+filename
                if full_filename.endswith(".txt"):
                    with open(full_filename, 'r', encoding='cp437') as f:
                        files[directory].append(f.readlines())

Thanks in advance


texts:
Im traying to integrate Hadoop and R, I was install the pachages rJava and Rhipe in R, I do this steps to start Hadoop and R:
-starting Hadoop services.,
-loading rJava and Rhipe packages by library function.
-Calling rhinit() to initialize Rhipe.
the problem here is when I call rhinit() funtion, it show this error:

    Initializing Rhipe v0.73
    Error in .jnew("org/godhuli/rhipe/PersonalServer") :
     java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream'

please some helps to fixe this problem.


texts: What clustering algorithm is appropriate for clustering paths?

texts:
I have a dataset with vectors in 2-dimensional space that form separate sequences (paths). Full data is presented below: ,
while a random sample of 5 paths looks like below (please note that incontinuity in paths are natural for the data and doesn't mean missing values):

I would like to find similar paths, where similar would mean (in order from the most to the less prominent):

they end up in a similar region
they are similar in direct length (i.e. length from start to end on x axis)
they are similar in complexity (i.e. number of vectors)
they starts in a similar region

What clustering algorithms are natural choice for such a setup? What are things worth to be aware while clustering paths? How can I deal with the fact, that different paths has different number of vectors? How can I represent a data to take that into account?


texts: Point of dropping weights in mini batch for purpose of regularization

texts: I have been reading "drop" is a method to regularize model better. It's purpose is to update only some % of weights in backprop and it helps you to not over fit the model. But I am wondering, is this method useful in minibatch at all?


texts:
My web application stores usage data, for example:

tickets opened an closed
tasks executed
user scores

etc. I need to show dashboards and reports for usage and performance trends, like:

How many tickets where opened/closed in a period?
what is the average tasks execution time?
who is the most active user?
which user gets the best score?

etc.
the set of questions is open, we can invent more.

Does this scenario fall under a datawarehouse classic approach?
Data are coming from a relational database with a specific data model.

Somebody is asking me to create a datalake, just giving him all my raw data.
I don't understand how does a datalake fit in this picture. With no model and relationship my data are usueless.

Also, I don't have a stream of events: I have tables in a relational database.
For example a row in a "task" table will have an "open time" column and a "close time" column.

I'm confused about the need or advantages of creating a datalake vs a traditional datawarehouse approach


texts: Data augmentation based on the class type in the CNN model

texts:
I would like to use CNN model to classify images but some classes in my dataset have low amount of data.

Can I apply data augmentation based on the number of the images in the class?

For example, the classes that contain 10 images will be 50 after augmentation that means the number of images is increased 500%, and the classes that contain 20 images will be increased 250% to be 50 images and the classes that contains 30 images will be increased 166% to be 50 images.

So, I would like to increase the data based on the number of images in the class.


texts: Keras: layer to obtain the combinations of rows in a Tensor of variable length?

texts:
I'm building a DNN architecture where I need to do independent operations on every unique pair of rows in a tensor in one branch, while the other branch computes a global representation that is concatenated to every pairwise-representation afterward.
For that, I want to compute a tensor of target_shape=(None, num_combinations, features) from a global tensor of original_shape=(None, num_rows, features)
So far, I've achieved this for constant-length targets while building the model (code to replicate the functionality below).
I´d like to know if it is possible to create a layer in Keras that can do this for a variable num_rows, taking an input of original_shape=(None, None, features) and outputting target_shape=(None, num_combinations, features).
I'm aware that I can use padding to get a fixed input size of maximum length, but in this case, it would be ideal to avoid that, because all the layers are shared for each row, so they don't care about the shape, and the axis "1" of my data has a considerable variance in size.
Current Implementation
from keras.layers import *
from itertools import combinations

example_tensor = K.constant([[[0.1, 0.11], [0.2, 0.22], [0.3, 0.33], [0.4, 0.44]]])
print("example:", K.eval(example_tensor))

len_tensor = 4  # I'd like this to be computed by the layer
n_features = 2  # Only 2 for the example,

picker = Lambda(lambda t, i: t[:, i, :], output_shape=(n_features,), name="picker")
reshape = Reshape((1, n_features), name="isolate_pair")
pair_merge_fnc = Multiply()  # for example, combine the features using element-wise multiplication

pair_indices = list(combinations(range(len_tensor), 2))
n_pairs = len(pair_indices)

pairs = []
for (a, b) in pair_indices:
    picker.arguments = {"i": a}
    e1 = picker(example_tensor)

    picker.arguments = {"i": b}
    e2 = picker(example_tensor)
    print("pair:", K.eval(e1), K.eval(e2))

    merged = pair_merge_fnc([e1, e2])
    merged = reshape(merged)
    pairs.append(merged)

pair_features = Concatenate(axis=1, name="stack_pairs")(pairs)
print("result:", K.eval(pair_features))

"""
OUTPUT

example: [[[0.1  0.11]
           [0.2  0.22]
           [0.3  0.33]
           [0.4  0.44]]]

pair: [[0.1  0.11]] [[0.2  0.22]]
pair: [[0.1  0.11]] [[0.3  0.33]]
pair: [[0.1  0.11]] [[0.4  0.44]]
pair: [[0.2  0.22]] [[0.3  0.33]]
pair: [[0.2  0.22]] [[0.4  0.44]]
pair: [[0.3  0.33]] [[0.4  0.44]]

result: [[[0.02       0.0242    ]
          [0.03       0.0363    ]
          [0.04       0.0484    ]
          [0.06       0.0726    ]
          [0.08000001 0.0968    ]
          [0.12       0.1452    ]]]
"""

I'm using Keras (2.2.4) with TensorFlow (1.13.1) backend


texts: Depth of the first pooling layer outcome in tensorflow documentation

texts:
Let's say that we have a CNN with two convolutional layers (https://www.tensorflow.org/tutorials/layers). My question regards the dimension of the tensor, which is the output of the pooling layer 1.

In the first convolutional layer, we apply $32$ filters to the input image (let's say that the output will be $28\times 28 \times 32)$, so as far as I can understand we will get $32$ separate feature maps, because of the number of filters.

In the next step, we can apply an activation function, which does not change the dimensionality.

The $\max$ pooling layer takes as input a tensor of $28\times 28\times 32$ and the output is going to be a $14 \times 14 \times \color{red}{1}$ tensor (according to the link above).

I cannot understand the unit as the depth, since we apply $32$ filters and we apply the $\max$ pooling layer to every feature map. So, why is the output tensor $14 \times 14 \times 1$?

According to my understanding the input in the second convolutional layer should be a $14 \times 14 \times 32$ tensor. Probably, I am missing something here.


texts: Is a multi-layer perceptron exactly the same as a simple fully connected neural network?

texts:
I've been learning a little about StyleGans lately and somebody told me that a Multi-Layer Perceptron, MLP, is used in parts of the architecture for transforming noise. When I saw this person's code, it just looked like a normal 8-layer fully connected network (i.e. linear-->relu-->linear-->relu-->...)
Last year, I read Hands-on Machine Learning with Scikit-Learn, Keras, and TensorFlow 2 by Aurelien Geron and he talks about MLPs. When I read about it, I interpreted his description as that an MLP is not exactly the same as a vanilla fully connected neural network. I didn't fully understand the text and don't have the book anymore so, unfortunately, can't recall exactly what I read so I might have been completely wrong in my understanding of what he wrote.
Is an MLP the same thing as very basic fully connected network?


texts:
I am working on spell checkers, I want to create a spell checker, I am confused about which model to use

Word-Level modelling
Character-Level modelling

plus I am preferring neural networks over Peter Norvig or N-gram/K-gram or any other "Vanilla" algorithm so that networks capture context

Example:
Input: I have aplied fo r credit cart
Output: I have applied for credit card

I have gone through models which use character level BLSTM with noise makers/error models and attention mechanism but not satisfied by result. https://towardsdatascience.com/creating-a-spell-checker-with-tensorflow-d35b23939f60

One of the article I have gone through is to use OpenNMT, but results were disastrous. https://medium.com/scribd-data-science-engineering/neural-spelling-corrections-and-the-importance-of-accuracy-977c0063d20f


texts: How to justify a predictor in influencing the outcome?

texts:
I am working on a prediction (binary classification) problem

Currently I get an AUC score of 85-86 and F1-score of 81

Questions

1) The above performance is based on 6 well-known features

2) Let's say I add another feature and I see my AUC and F1 score improve by 1 or 2 points? So is this the only way to know that it really helps/value addition to the model?

3) How do I justify that these features really help in predicting the output?

4) Is there anyway to prove or validate that adding this feature really helps my model and improves the outcome? Is it only using changes in AUC score?

5) For instance, I can add multiple features and my AUC increases by few decimal points, so can I say that they are important or useful or drive predictions? Yes ofcourse, but adding 10 features (where 6 features really impact the outcome and rest 4 increase the auc only by few decimal points). doesn't really overfit. Am I right? Because my prediction score doesn't go beyond 86


texts:
I have an understanding problem. I am a beginner in machine learning and have also a little experience in modelling NNs but not for time series.

But I cannot imagine how to use Neiural Networks for time series.

So if I want to train a Multilayer NN what is the input?

I read several papers about such ANN to predict time series. But they did not explain how actually.

Can I imagine it as follows: Is every input node a specific time stamp?

If I want to predict the value for time t+1, and I use 15 input neurons, so I use the values at each time stamp from t-15 upt to t?

How do I train such a NN?

I am a little confused.


texts: GPU memory requirement calculation from network architecture

texts:
I would like to get a rough idea of the GPU Ram requirement for a network based on the number of layers and the number of nodes at each layer.  Is it
number of nodes * 2 * datatype size (64 for double) + number of connections * 2 * datatype size

?
(Using PyTorch)


texts: What's a difference between the neoperceptron and CNN?

texts:
What's a difference (in terms of architecture) between the neoperceptron and CNN?

Both ANNs have hidden layers and scanners, as I understood, but many sources subdivide them in two classes.


texts: Unsupervised Learning::Satellite Images::Single Bands

texts:
Has anyone has success with building models using KMeans for classification? I have images that only have one band and it continues to fail. My guess is that the issue is with both size of the image as well as the single band.

For example:

from osgeo import gdal,gdal_array
import numpy as np

src = '/Path/ImgA.TIF'
img_A = gdal.Open(src)

#Getting bands (count)
bands_n = img_A.RasterCount  #returns 1

band = img_A.GetRasterBand(1)

#read as array
band_arr = band.ReadAsArray()

band_sh = band.shape

#eg. output#
300,300

//This is where I am getting stuck. If I pass only two inputs (rows/cols) to KMeans, it fails as it requires a 2D array not 1D. It also fails when I manually set the band://

#Attempt 1
rows, cols = band_sh

#Attempt 2
rows, cols, band = band_sh, 1

X = band.reshape(rows*cols,band) // X = band.reshape(rows*cols,1)

from scikit.cluster import KMeans

kmeans = KMeans(n_classes = 2, random_state=2).fit(X)

Any ideas? This works just fine with RGB but fails each time when dealing with rasters that are single band.


texts: What is the problem with this architecture ? I am getting error negative dimensions. I want to avoid dense layers and dropouts

texts:
model = Sequential()
model.add(Conv2D(64, (1, 1), activation='relu', input_shape=(64,64,3)))
model.add(Conv2D(64, (5,5), strides=(2,2), padding='same'))
model.add(MaxPooling2D(pool_size=3, strides=(2,2)))
model.add(Conv2D(64, (1, 1),
                          strides=(1,1),
                          activation='relu',
                          padding='valid' ))
model.add(Conv2D(64, (3, 3),
                          strides=(1,1),
                          activation='relu',
                          padding='valid' ))
model.add(Conv2D(64, (1, 1),
                          strides=(1,1),
                          activation='relu',
                          padding='valid' ))
model.add(Conv2D(64, (3, 3),
                          strides=(1,1),
                          activation='relu',
                          padding='valid' ))
model.add(Conv2D(64, (1, 1),
                          strides=(1,1),
                          activation='relu',
                          padding='valid' ))
model.add(Conv2D(64, (3, 3),
                          strides=(1,1),
                          activation='relu',
                          padding='valid' ))
model.add(Conv2D(128, (1,1),
                strides=(2,2),
                activation='relu',
                padding='valid'))
model.add(Conv2D(128, (3,3),
                strides=(1,1),
                activation='relu',
                padding='valid'))
model.add(Conv2D(128, (1, 1),
                          strides=(1,1),
                          activation='relu',
                          padding='valid' ))
model.add(Conv2D(128, (3,3),
                strides=(1,1),
                activation='relu',
                padding='valid'))
model.add(Conv2D(128, (1,1),
                strides=(1,1),
                activation='relu',
                padding='valid'))
model.add(Conv2D(128, (3,3),
                strides=(1,1),
                activation='relu',
                padding='valid'))
model.add(Conv2D(128, (1,1),
                strides=(1,1),
                activation='relu',
                padding='valid'))
model.add(Conv2D(128, (3,3),
                strides=(1,1),
                padding='valid'))

model.add(AveragePooling2D(pool_size=2, strides=(2,2), padding='valid'))
model.add(Conv2D(1,1,200))
model.add(Flatten())
model.add(Activation('softmax'))


texts: Same input size but cannot fit the model in keras

texts:
I am trying to fit this model in keras but getting this error :

 Namespace(batch_size=32, epoch=10, num_classes=2)
Start
Train loading
<class 'numpy.uint8'>
Test loading
<class 'numpy.ndarray'>
Validation loading
done_loading
Traceback (most recent call last):
  File "main.py", line 67, in <module>
    validation_data=(x_val, Y_valHot)
  File "/tools/anaconda3/envs/py35/lib/python3.5/site-packages/keras/engine/training.py", line 1581, in fit
    batch_size=batch_size)
  File "/tools/anaconda3/envs/py35/lib/python3.5/site-packages/keras/engine/training.py", line 1414, in _standardize_user_data
    exception_prefix='input')
  File "/tools/anaconda3/envs/py35/lib/python3.5/site-packages/keras/engine/training.py", line 153, in _standardize_input_data
    str(array.shape))
ValueError: Error when checking input: expected data to have shape (None, 224, 224, 3) but got array with shape (9730, 244, 224, 3)

The function used to fit the model is

model.fit(X_train, Y_train,
          batch_size=batch_size,
          nb_epoch=nb_epoch,
          shuffle=True,
          verbose=1,
          validation_data=(X_valid, Y_valid),
          )

Can someone tell me where I went wrong.


texts:
Say I have an intermediate embedding layer (say at first layer) in a neural network of four layers. A desired property of the embedding matrix is that the sparser the better. Say the output from neural network is o1, is it ok to make the score like
score(o1)+sparse(embedding matrix)? That is we not only use the final output, also the intermediate embedding layer (intermediate output)?


texts:
I am reading, a machinelearningmastery.com post about improving neural network performance, and I am attempting to normalize my own data set with the tips mentioned in the post using sklearn preprocessing MinMaxScaler.

In my code I am reading a CSV file directly into pandas.

#read CSV file
df = pd.read_csv('C:\\Users\\desktop\\EC\\data.csv', index_col='Date', parse_dates=True)

The snip below is what the data looks like with column names:

From the post, I normalize the data with this code below. The process appears to just return a numpy array, but I use Pandas during the machine learning fit process.

from sklearn.preprocessing import MinMaxScaler

# create scaler
scaler = MinMaxScaler()

# fit and transform in one step
df2 = scaler.fit_transform(df)

df2 = pd.DataFrame(df2)

What's happening, is my column names are stripped away and I use column names a lot in dropping & selecting. For example, I fit a lot of models like this process below to differentiate target and input variables.

#Test random Forest

import numpy as np
from sklearn import preprocessing, cross_validation, neighbors
import pandas as pd
from sklearn.ensemble import RandomForestRegressor
from sklearn.externals import joblib
import math

rmses = []
for i in range(2):

    X = np.array(df2.drop(['Demand'],1))
    y = np.array(df2['Demand'])

    offset = int(X.shape[0] * 0.7)
    X_train, y_train = X[:offset], y[:offset]
    X_test, y_test = X[offset:], y[offset:]

    clf = RandomForestRegressor(n_estimators=120, min_samples_split=20)

    clf.fit(X_train, y_train)
    mse = mean_squared_error(y_test, clf.predict(X_test))
    rmse = math.sqrt(mse)
    print("rmse: %.4f" % rmse)
    rmses.append(rmse)

print(sum(rmses)/len(rmses))

joblib.dump(clf, 'rfrModel.pkl')

But maybe this isn't a bid deal... Sorry not a lot of wisdom here any tips help. Is there another method to madness that I don't need to dependent on column names? If the df2 = scaler.fit_transform(df) leaves columns in place but just removes names, I could just use a column number to .drop - differentiate target & input variables.. Demand is the name of my target variable, and I could just call the second column..., right??


texts: What is Difference Between np.zeros() and np.empty()

texts: Can Anyone Please Explain me the Difference Between them In Terms Of Any Operation or Computation and Values Stored in them


texts: a simple way to test wether a tree-based classifier would transfer well to a target population?

texts:
I trained a tree-ensemble classifier (XGBOOST) on population A, validated it and I'm satisfied with its accuracy (AUC 0.78). Now I'm trying to transfer it to a slightly different population B, and there the accuracy of the model deteriorates badly (AUC 0.68)

I tried isolating which of the features did not transfer well, both by simple univariate analysis (comparing distributions), and by comparing each feature correlation with the label, and couldn't find anything obvious.

Is there a way to debug and understand which of the model assumptions which held at A do not hold at B? I thought about comparing the label distributions at every node in every tree for the validation populations in A and B, thereby testing all conditional probabilities the model assumes are holding actually hold at B.

Would that help me understand what broke? or would I just get tons of tiny differences? Is there some other simple way I'm missing?

(related to this survey Machine learning learn to work well on future data distribution?)


texts:
According to Wikipedia, "the distance matrix of size $\frac{(n^2-n)}{2}$ can be materialized to avoid distance recomputations, but this needs $O(n^2)$ memory, whereas a non-matrix based implementation of DBSCAN only needs $O(n)$ memory."

$\frac{(n^2-n)}{2}$ is basically the triangular matrix. However, it says that a non-matrix based implementation only requires $O(n)$ memory. How does that work? Regardless of what data structure you use, don't you always have to have $\frac{(n^2-n)}{2}$ distance values? It would still be $O(n^2)$ space complexity, no? Is there something I'm missing here? I'm working with a huge dataset and I would really like to cut down on memory usage.


texts: Using Image Data along with CSV file data for CNN model

texts:
I have image data along with csv file where each row of csv file contains attributes for corresponding image.

I want to use images as well as csv file data to build CNN model using Keras.

What is preferred way of doing it?

I can append the attribute data with numpy array of image and train the model.
I can create 2 different models and use average/voting for prediction.

I think, in 1st appraoch csv file data which has around 50 columns would hardly matter as image shape is (128*128*3). Is it correct? and how can I give more importance to csv file (like 60% weightage to image data and 40% to csv file).

And again how CNN would work with that CSV file data as CNN is preferred on Image Data?


texts: Are there any python libraries for sequences clustering?

texts: I have a problem which I explained in other question. I've understood that my dataset is a sequence of states or something like that. Is there libraries to analyze sequence with python? And is it right way to use Hidden Markov Models to cluster sequences?


texts: How Does Sci-Kit Learn Train Regression Neural Networks (MLPRegressor) So Fast?

texts: Why does using the scikit-learn library's MLPRegressor result in such a boost in training time when compared to constructing the network from scratch? I tried both methods and I found that writing the code from scratch yielded an average training time of ~10 seconds while Sci-Kit Learn trained almost instantly?


texts: Multivariate time Series classification - One class

texts: I need your help with time series classification. I have measurements of different medical parameters for patients captured at every one hour. The output label is whether the patient has Acute Kidney Injury(AKI) or not. Based on the first 12 hour data, we should find out whether the patient has the risk of suffering from AKI or not (After 12 hours). I guess this falls under classification approach (Sequence Classification). However I have only one label (AKI == 0). So should this be considered as Anomaly detection in Time series or Sequence classification?  Since I have more than 100 patients data for 12 hour (100 * 12 datapoints with multiple input variables), how do I retain the time factor? As there is only one class, how do I do the training? I am quite stuck as I am quite new to this area. Can you please share your insights/ guide me as to how to approach this problem/direct me to the appropriate resource/sample project?


texts: Partial Dependence Plot and categorical variables

texts:
While reading about machine learning explainability and Partial Dependence Plot (PDP) in this book, the following appeared when dealing with categorical variables:

For each of the categories, we get a PDP estimate by forcing all data instances to have the same category.

Is there a better explanation of how to deal with categorical features in PDP?

How to do correctly Partial Dependence Plots with categorical variables


texts: Using Keras fit_generator for functional keras models and custom dataset

texts:
I have to fit a model that takes three discrete inputs and produces two discrete outputs using a generator made as follows:

class Dataset(tf.keras.utils.Sequence):
    def __init__(self, text):
        ...

    def __len__(self):
        return len(self.text)

    def __getitem__(self, item):
        data = process_data(self.text[item])

        feat1 = np.array(data["ids"], dtype=np.long)
        feat2 = np.array(data["mask"], dtype=np.long)
        feat3 = np.array(data["att"], dtype=np.long)

        targets = np.array(data["offsets"], dtype=np.long)
        return (feat1, feat2, feat3), (targets[:,0], targets[:,1])

And using this dataset, if I want to fit the model that takes three inputs and generates two outputs (Something like shown below):

How should I fit this generator into Keras fit_generator function?
Which should be something like this:

train_dataset = Dataset(df.text_train)
valid_dataset = Dataset(df.text_valid)

model.fit_generator(generator=train_generator, valid_data=valid_generator)

Right now, this is throwing error with the tuple from the generator stating tuple has no attribute shape.


texts: how to generate sample dataset for classification problem

texts: I am a newbie to data science. I have a 'short text' categorization problem where input variables are either unstructured texts (names, definition, description etc) or categorical. There is not much semantic to the fields as they are product names, territory name, sales order type etc. Issue is I do not have any sample data set from which I can derive training, test, validation set or divided it into k-fold for cross validation. So how should I generate sample data? I have about 20 target classes. I can classify some dataset using regex or lucene rule based matches and manually verify them and make sure each class have equal amount of samples. But I am open to other suggestion.


texts:
I actually need to decide on which computer I will be taking for a Data Sciences curriculum, including machine learning and further hands on Hadoop. Some computer vision applications are also planned.
I already have a Tesla K20 5GB, I think it's enough for beginning, they suggest me the following choices :
Choice 1 : Core i5-4570n 16GB RAM, 240GB SSD+3TB HDD
Choice 1 : Ryzen 5 2600, 16GB RAM, 480GB SSD+3TB HDD

What would you recommend me ? Some changes can be done over configurations if necessary.


texts: When to use Linear Regression and When to use Logistic regression - use cases

texts:
Linear Regression is used to find the relation between dependent variable and independent variable.

Logistic Regression is used when dependent variable is categorical.

But this doesn't exactly suffice, what use cases you can use either of these, or can we use a combination of them?


texts: BinaryEncoder on test set causes different feature count than on training set

texts:
I am using a BinaryEncoder from the category_encoders package. I used it to encode a feature in my training set: 30k samples, 1170 unique categories for the feature in question, that resulted in 12 new features. I am trying to apply the same encoder to the test set (5000 samples, 603 unique categories), that resulted in 11 new features. The problem is that now the fitted model cannot predict on the test set because the dimensions do not match (36 total features vs 35 in the test set).
Edit: the test set is a separate file on which I have to do predictions, not part of the training set split by me.
Input 0 of layer dense is incompatible with the layer: expected axis -1 of input shape to have value 36 but received input with shape (None, 35)
What am I doing wrong?


texts: How Dummy Variables Should Be Modeled In A Linear Regression Model?

texts: I've a cross sectional model where I want predict number of users that take specific service, to make it I've many variables but have specifically two nominal: isWorkday(0 or 1) and weeday(1,2,3,...,7). When I make the model, taking into account the two variables, generates high multicollinearity. So I've delete one of them, so what's better have many dummies (weeday) or less dummies (isWorkday).


texts: Python Pandas - Interpolation with fixed value

texts:
Say we have a pandas series with the following values

[np.nan, np.nan, 1, np.nan, 2, np.nan]

What is the most efficient way fill the nan value with 0 in the middle. so we have

[np.nan, np.nan, 1, 0, 2, np.nan]

In other word, how to we do interpolation with a fixed value, or a .fillna operation but ignore the nan at the beginning and end of the array.


texts: Binary classification as a 2-class classification problem

texts:
I want to create a dog-classifier, which outputs the probability of an image containing a dog.

I have two approaches in mind -

Binary classifier (1-class), which just outputs the probability of the image containing a dog. This seems reasonable to me.
2-class classifier with two classes denoting "dog" and "not-dog". But my problem with this approach is that the neural network has to learn the "not-dog" class as well, which is impossible since it has no pattern and is different in each training example.

Would the second approach be less effective than the first? Or even work at all?

Thank you.


texts: For obejct detection, should I resize my custom images first and then start the annotation or it won't matter?

texts: I have my custom dataset images of size (1080 x 1920) and I am trying to use yolov3 for object detection. I noticed that yolov3 model accepts an input image size of 416 x 416. So I am in confusion if I should resize the image and apply zero-padding to save the aspect ratio and start my annotation after that OR should I annotate my custom images in original size? And will data argumentation affect the annotation while training? Thanks


texts: Using an unsupervised Isolation Forest, how does one identify the optimal number of outliers from the anomaly scores?

texts: I am using an unsupervised isolation forest algorithm and computing anomaly scores to detect outliers from a 2 dimensional toy dataset. From a scatter plot, I am able to detect/visualize the data points with the highest anomaly scores (example: top 10 or top 15 outliers from the data)as my outliers. Is the number of outliers subjective to user decision (for example:Anomaly score of 0.5>S>1 is an outlier and everything less than 0.5 is not) or is there a way to detect the optimal number of outliers based on the anomaly scores?


texts: Can points not being passed to the loss function influence it?

texts:
If I were to construct a model where only a subset of the training data is passed to the loss function, can the other parts of the dataset influence the fit?
More concretely, in this first example:
model.fit(x_train_subset, y_train_subset, val=[x_val, y_val])

Here, I am passing the model, and then the loss function, only the subset to be trained on.
In another example:
model.compile(loss=CustomLoss) #this loss will be based on x_train_subset
model.fit(x_train, y_train, val=[x_val, y_val])

Here, I have passed the entire training set to the fit, but using only a subset of them to find the loss.
My questions are:

Conceptually, are these the same?
If a point is not being passed to the loss function is it effectively invisible to the model training procedure?


texts: Problem in working out an example of SVM: mathematical steps

texts:
I am trying to code SVM from scratch using a small toy problem that involves five support vector values. In the code below, there are 5 support vectors arbitrary chosen and denoted by the variables s1,s2,s3,s4,s5. The support vectors are augmented with a third coordinate which is the bias = 1. y denotes the labels for the 3 support vectors. A is the 5 by3 design matrix that contains the values:
11  0   5
22  0   5
33  0   5
22  8   5
33  8   5

Thus the equation becomes y = wx + b where x is the input data. The equation for the hyperplane is w = sum_i a_i*s_i where the a_i's are the alpha parameter for i=1,2,3,4,5.
Confusion: Should I take the transpose of A in the least squares solution: alpha = y/A' and wouldn't there be 5 values for alpha and hence w = [alpha1*s1+alpha2*s2+alpha3*s3 + alpha4*s4 alpha4*s5 ]? But I am getting 3 values for alpha instead of 5. Is it because there are 3 coordinates or somewhere the product is becoming zero?
% 5 support vector
s1 = [1 0 1];
s2 = [2 0 1];
s3 = [3 0 1];
s4 = [2 2 1];
s5 = [3 2 1];

s_x = [1 2 3 2 3 ];
s_y = [0 0 0 2 2];

y = [-1 -1 -1 +1 +1]

gscatter(s_x,s_y,y)

A = [ (s1.*s1)+ (s2.*s1)+ (s3.*s1) + (s4.*s1) + (s5.*s1);
      (s1.*s2)+ (s2.*s2)+ (s3.*s2) + (s4.*s2) + (s5.*s2);
      (s1.*s3)+ (s2.*s3)+ (s3.*s3) + (s4.*s3) + (s5.*s3);
      (s1.*s4)+ (s2.*s4)+ (s3.*s4) + (s4.*s4) + (s5.*s4);
      (s1.*s5)+ (s2.*s5)+ (s3.*s5) + (s4.*s5) + (s5.*s5);]

  alpha = y/A'
  alpha1 = alpha(1)
  alpha2 = alpha(2)
  alpha3 = alpha(3)

  % w= sum a_i*s_i
  % y = wx +b
  w = [alpha1*s1+alpha2*s2+alpha3*s3 ]


texts:
I'm trying to get a NN (a CNN, to be precise) to predict labels on hugely imbalanced data, i.e. some classes are much bigger than others. I want to optimize the balanced accuracy. So I give the class weight to the model which I compute like so:
classes = np.unique(y_ord)
class_weight = compute_class_weight('balanced', classes=classes, y=y_ord)
class_weight = {k: v for k, v in zip(classes, class_weight)}  # make class_weight a dict

Now, I train my model using the sparse_categorical_crossentropy as the loss. However, while the loss decreases, accuracy does not improve. I've read in several blog posts the (possible) reason why: The loss is continuous, while the accuracy is either 0 or 1. However, I've failed to find a way to deal with this. This does not happen when I do not use the weights, so I guess it must have something to do with these.
Thank you in advance!


texts: Different performance for splitting into test/train data vs. using cross-validation

texts:
I am training a linear model using the following scikit-learn setup:
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.svm import LinearSVC
from sklearn.metrics import f1_score
from sklearn.model_selection import cross_val_score

[...]

random_state=786543

max_iter=5, tol=None)
clf = LinearSVC(random_state=random_state, dual=True, C=1.5)

X_train, X_test, y_train, y_test, i_train, i_test = train_test_split(feature_matrix, y, indices, test_size=0.33, random_state=random_state)
clf.fit(X_train, y_train.values)
predicted_train = clf.predict(X_train)
predicted_test = clf.predict(X_test)
print('Train Accuracy: ' + str(np.mean(y_train == predicted_train)))
print('Test Accuracy: ' + str(np.mean(y_test == predicted_test)))
print('Test F1 micro: ' + str(f1_score(y_test, predicted_test, average='micro')))
print('Test F1 macro: ' + str(f1_score(y_test, predicted_test, average='macro')))
print('Test F1 weighted: ' + str(f1_score(y_test, predicted_test, average='weighted')))

Train Accuracy: 0.985129495926343
Test Accuracy: 0.9601936525013448
Test F1 micro: 0.9601936525013448
Test F1 macro: 0.9000889214688401
Test F1 weighted: 0.9590331562500389

But now I run
scores = cross_val_score(clf, feature_matrix, y, cv=5, scoring='f1_macro')
print(scores)

array([0.65860981, 0.84306338, 0.82113645, 0.83414211, 0.64665942])

How can this discrepancy be explained? I tested this using different random states.
A couple of points to consider:

I am having multible classes (but only one label per sample)
The dataset is skewed (so there are classes with many samples and some with very few classes)
I have 45066 samples, 5222 features, 259 classes

The number of samples per class is:
sorted(list(np.unique(y, return_counts=True)[1]))

[1, 1, 1, 1, 1, 1, 1, 2, 3, 4, 4, 4, 4, 4, 7, 7, 8, 9, 9, 10, 10, 10, 10, 10, 10, 10, 10, 10, 11, 11, 11, 11, 11, 11, 11, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 13, 13, 13, 13, 13, 13, 14, 14, 14, 14, 14, 14, 15, 15, 15, 15, 15, 16, 16, 16, 16, 16, 17, 17, 17, 17, 17, 18, 18, 18, 19, 19, 19, 19, 19, 19, 20, 20, 20, 20, 20, 20, 21, 21, 21, 22, 22, 22, 22, 23, 23, 24, 24, 24, 25, 25, 25, 26, 26, 27, 27, 27, 27, 27, 29, 29, 29, 29, 30, 30, 30, 30, 32, 32, 32, 34, 34, 35, 35, 35, 36, 36, 36, 36, 36, 36, 37, 37, 37, 37, 38, 38, 38, 38, 40, 40, 40, 41, 41, 45, 45, 45, 46, 46, 46, 46, 47, 47, 47, 48, 49, 50, 50, 52, 55, 56, 59, 59, 60, 60, 61, 61, 61, 65, 65, 67, 67, 69, 72, 73, 74, 75, 77, 77, 79, 80, 84, 85, 87, 93, 96, 97, 97, 103, 110, 112, 117, 123, 130, 139, 139, 141, 143, 146, 146, 147, 147, 150, 159, 161, 169, 170, 177, 180, 180, 189, 191, 196, 198, 199, 201, 202, 203, 203, 208, 211, 230, 236, 249, 255, 264, 268, 269, 300, 332, 347, 356, 358, 364, 388, 433, 469, 476, 484, 548, 652, 698, 723, 748, 753, 807, 815, 1013, 1200, 1222, 1243, 1274, 1447, 1643, 1741, 2900, 3909, 4627]


texts:
I have a bit of self taught knowledge working with Machine Learning algorithms (the basic Random Forest and Linear Regression type stuff). I decided to branch out and begin learning RNN's with Keras. When looking at most of the examples, which usually involve stock predictions, I haven't been able to find any basic examples of multiple features being implemented other than 1 column being the feature date and the other being the output. Is there a key fundamental thing I'm missing or something?

If anyone has an example I would greatly appreciate it.

Thanks!


texts:
I am working on a personal loan dataset. For each loan, we recorded its credit status monthly after it was drawn by the borrower. Let's say there were 6 status coded by A-F. My project is to use Markov chain model to train the data and estimate the transition matrix as shown below. Then we can predict the future movement of any single loan in probability.

Meanwhile, the dataset contained substantial features for a single loan, like loan amount, borrower age, income, dwelling region, bank account profile, last 90 days bank statement data, some other credit bureau data etc.

I just get this project from the very beginning and all thoughts are rough but not accurate. I need brain storm. Thanks heaps.

How can I use the data properly to predict the next state given the current state? Any assumption should be seriously considered in this project and any advice or experiance for me? I will do it in Python and/or R.


texts:
I am making a project on prediction cars price given its features. I was able to scrape over 13000 examples. After cleaning and manipulating the data, I left with a little more than 11000 examples, I used 12 features of car, like mileage, year, brand and all other important stuff. After model selection, I decided to use Random forests to predict model. There are several questions, that got me interested:

1) When tuning model, the best I can get on test set is mean absolute error more than 2200. R_2 score is 0.92
And training set error is 900-1000. R_2 score is 0.98
I couldn’t eliminate this overfitting, if it was, what I think it is.
I tried grid search on bagging, forests even, on boosting (I know it reduces bias, not variance, but I was desperate) with different parameters, but the test error and train error with best parameters were always approximately same.
I used simple estimators too, but they had too much bias in train and test.
I know that there is a way to get rid off this overfitting, if only I had more data, but this data is maximum I can get from the page I scraped.
So the first questions is: are there situations when overfitting cannot be solved without proper and big dataset? Am I stuck in this situation?

2) And the second optional question is:
Some of the features in data are categorical, so after creating dummy variables I had something like 120 features. And interesting thing is, that after scaling, which I needed for PCA, the variance I retained with arbitrary n_components was completely different, than variance I retained from data scaled just on continuous features, and not on dummy. I know that there is no meaning in scaling dummy variables, and it’s not handful. But after scaling dummy variables, in order to retain 99 percent of variance, I needed something like 110 features (originally 120), and after scaling just continuous features, to retain 99 percent of variance I needed less than half of the features (~55).
It’s strange for me, I cannot understand this behavior, is it okay at all?


texts: I am new to Statistics and was reading about Decision errors in Hypothesis Testing. My question is that why is Type-II error an error at all? From what I understand, it arises when we fail to reject a false null hypothesis. When we fail to reject null hypothesis, it simply means that we do not have strong evidence to reject it. We are not making any comment about which of the two hypothesis is true (or false) . Either can be true. We are not saying that the null hypothesis is true. Then, why is such a conclusion called an error?


texts: CSVs: how to output missing data to make processing easier?

texts:
If supplying data (that can be either string or numeric) via CSVs, what's a good strategy for marking that a value is missing?

Some non-empty sentinel, like NA
Some non-empty sentinel, but quoted like "NA"
The empty string, e.g. in the CSV the value will be 0 characters long.
The quoted empty string ""
Something else / depending on case?

If it makes a difference, would like the CSVs to behave reasonably when read using R readr, Python Pandas, and Excel.


texts: Pandas column count mismatch after insert into dataframe

texts:
I am trying to insert a column of values from one dataframe to another. The total count of values of both the indiviual dataframes is 91744 but when I use

df1.insert(loc=0, column='x1', value=ts)

to insert the values of ts as the first column in df1, the value count after insertion turns out to be 84175 instead of actual 91744. Please tell me what is wrong with this.

>>> df1.insert(loc=0, column='x1', value=ts)
>>> df1.x1.count()
84175
>>> ts.count()
0    91744
dtype: int64

>>> df1.describe()
                 x1            x2
count  84175.000000  91744.000000
mean       0.249044     11.744387
std        0.191239     13.294185
min        0.000000      1.000000
25%        0.108383      3.000000
50%        0.212760     12.000000
75%        0.311860     15.000000
max        1.000000    551.000000

Sample data is given below.

>>> df1.head()
   x2  x3  x4   x5                   x6  y
0  12   1  13  497  2017-11-07 09:30:38  0
1  25   1  17  259  2017-11-07 13:40:27  0
2  12   1  19  212  2017-11-07 18:05:24  0
3  13   1  13  477  2017-11-07 04:58:08  0
4  12   1   1  178  2017-11-09 09:00:09  0
>>> ts.head()
         x1
0  0.218561
1  0.269357
2  0.257698
3  0.238417
4  0.164645

My objective was to add the x1 column from of ts dataframe as the first column of df1 data frame.


texts: Deciding whether to train a saved model or start a new one in Keras?

texts:
I'm using Keras for various machine learning projects and save my models after every training run. Let's say I want to do a training session for an algorithm - how do you decide whether to (1) load a saved algorithm and train it or (2) start training the algorithm from scratch? Some potential approaches I thought of:

Only load a saved model when the new, labeled, examples differ significantly from what the saved model was trained on
Always train a new model since data is data
Always train a saved model since even small sampling variations can have an effect on the algorithm
Always train a saved model as soon as the input & output shapes are determined; you should only be training a model from scratch once.


texts: Tips on how to create box whisker plot for huge data set

texts:
I have a huge dataset (about 2 million lines) that I want to visualize to have an idea of how spread the data is. The problem now is when I create a box whisker plot the resulting graph is not legible due to the huge amount of data.
Is there any trick to be able to successfully create a box whisker plot for huge data set in the way that is readable?


texts: Best approach for classification problem where examples most belong to one set

texts:
I'm working to build an opt-out filter for my company. I have a small amount of machine learning experience (I've done a few projects with tensorflow in the past), but wanted to get other opinions on the best approach here.

My problem

We have a small dataset of messages (a few hundred) that have been labeled as "opt-out". They are all pretty similar in content, but different enough that we don't just want to do a fuzzy SQL query. The following are examples of messages that we want to label as "opt-out":

"Not interested. Please stop texts"
"Pls stop texting"
"Not real sure how you got my number... but I am not interested"

On the other hand, >99% of our messages are ok and should not be filtered. There are other interesting cases that I'd like the algorithm to pick out, for example:

"Can you stop texting me and email me instead" is actually clearly not an opt-out.

Question

From the reading I've done so far, it sounds like a RNN with LSTMs are the way to go, but I have very little understanding of these (yet) beyond the basic concept.

I'm interested in any thoughts or insights as to how I might proceed in this specific case where the size of negative examples is so much smaller than the set of positive examples. Would this aspect change the way in which is should train or evaluate my data?


texts: hmmtrain in Matlab converging to different values of trans and est over different runs

texts:
As the title indicates, over multiple runs of my program, I get different values of trans and est. Are these local minimum? If so how do I get the optimal one?

NOTE - They all have the same starting value of trans and est.

Thanks!

Edit: I realized why I was getting different values of trans and est over different runs but it raised another question in my mind.

My objective is to train an HMM on classical song data (MFCC coefficients). To discretize it, I assigned the MFCC coefficients to 7 clusters.

I have a sequence of 10 observations stored in matrix M that I am providing to the hmmtrain function as input. Each row represents a song, and each column are the different clusters the MFCC coefficients (for different frames of the song) were assigned to.

ex:

M =

 1     1     1     3     3     3     3     2     7     7     7
 1     1     1     3     1     1     1     1     3     1     1
 3     3     3     2     2     3     3     2     2     3     3
 6     6     6     2     2     5     5     2     2     5     5
 3     3     3     3     1     1     1     1     1     1     1
 7     7     7     7     5     5     5     7     2     5     5
 3     3     3     2     2     3     3     2     2     6     6
 7     7     5     7     2     3     6     2     2     2     2
 7     7     7     7     7     7     7     7     7     7     2

However, since I was recomputing the clustering every time, my input data would change. By saving the matrix to a file and using that every time my results were consistent. What I do not understand is why for certain formations of clusters (i.e. for certain input matrices) I would get better performance? Shouldn't cluster formation be the same by and large (even if the number ex. 1,2.. might represent a different cluster)?

Sorry about earlier. I'll repost the question if necessary. Thanks again!


texts: Tensorflow, Optimizer.apply_gradient: 'NoneType' object has no attribute 'merge_call'

texts:
My program gives the following error message:
 AttributeError                            Traceback (most recent call last)
<ipython-input-56-8f384e36cbe9> in <module>
     49         print(loss.numpy())
     50         grads = tape.gradient(loss,g.trainable_variables)
---> 51         optimizer.apply_gradients(zip(grads,g.trainable_variables))
     52         values.append(value.numpy())
     53     print('... Value =',values[-1])

~/.local/lib/python3.6/site-packages/tensorflow/python/keras/optimizer_v2/optimizer_v2.py in apply_gradients(self, grads_and_vars, name)
    437       self._prepare(var_list)
    438
--> 439       return distribute_ctx.get_replica_context().merge_call(
    440           self._distributed_apply,
    441           args=(grads_and_vars,),

AttributeError: 'NoneType' object has no attribute 'merge_call'

The interesting lines are:
epochs = 20
batch_size = 100
learning_rate = 0.01
g = G()
optimizer = Adam(learning_rate)
values = []
matrix = tf.constant(np.tri(NT+1),dtype=tf.float32)
for epoch in range(epochs):
    print(' Epoch',epoch+1)
    batches = dataset.shuffle(sample_size).batch(batch_size,drop_remainder=True)
    for batch in batches:
        with tf.GradientTape() as tape:
            g_values = tf.reduce_sum(g(batch),2)
            g_integrals = tf.tensordot(g_values,matrix,[[1],[1]]) * T/(NT+1)
            f_values = f(batch)
            # print(f_values.numpy())
            integrand = f_values * tf.exp(-g_values)
            value = tf.reduce_mean(integrand) * T + tf.log(2.)/alpha
            loss = -value
        print(loss.numpy())
        grads = tape.gradient(loss,g.trainable_variables)
        optimizer.apply_gradients(zip(grads,g.trainable_variables))
        values.append(value.numpy())

I do not see the point. f_values, integrand, loss, grads all contain no inf or nan, just ordinary tf.float32 numbers, idem g.trainable_variables. On another PC, the same code works fine. I use TensorFlow, see:
import numpy as np
import tensorflow as tf
tf.enable_eager_execution()
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Dense,BatchNormalization
from tensorflow.keras.optimizers import Adam

(The tf.enable_eager_execution command is set because, for a reason I do not know, "pip install tensorflow==2.0.0-beta1" only installs tensorflow 1.4.0, according to
print(tf.__version__)
.)
The network architecture is as follows:
class G(Model):
    def __init__(self,depth=2,hidden_units=32,dim=dim):
        Model.__init__(self=self)

        self.depth = depth

        self.mylayers = []
        for l in range(depth-1):
            self.mylayers.append(Dense(hidden_units,activation='tanh', \
                name='Layer{}'.format(l+1)))
        self.mylayers.append(Dense(1,activation='sigmoid',name='Layer{}'.format(depth)))
        self.BN = []
        for l in range(depth):
            self.BN.append(BatchNormalization())

    def call(self,inputs):
        data = inputs
        for l in range(self.depth):
            data = self.BN[l](data)
            data = self.mylayers[l](data)
        return data

Your help would be much appreciated.


texts: RandomizedSearchCV() not scoring all fits

texts:
I'm experiencing an issue with a RandomizedSearchCV grid that is not able to evaluate all of the fits. 50 of the 100 fits I'm calling do not get scored (score=nan), so I'm worried I'm wasting a bunch of time trying to run the gridsearch. I'm wondering how to troubleshoot this and haven't found anything in the past few days and I'm hopeful that the community can help me squash this bug. Now, the details:
I have constructed a XGBClassifier model as such:
xgb_clf = xgb.XGBClassifier(tree_method="exact", predictor="cpu_predictor", verbosity=1,
                            objective="binary:logistic", scale_pos_weight= 1.64)

# my trainingset is imbalanced 85k majority class, 53k minority class

Currently, I am attempting to use the hashing trick to encode my categorical variables, as they are all nominal. I do this after splitting my training set into X and y variables
ce_hash = ce.HashingEncoder()
hashed_new = ce_hash.fit_transform(X)
hashed_X = hashed_new

I then conduct my train_test_split as normal, then instantiate a RandomizedSearchCV with a parameter grid, code is as such:
X_train, X_test, y_train, y_test = tts(hashed_X, y, test_size=.25)

# create my classifier
xgb_clf = xgb.XGBClassifier(tree_method="exact", predictor="cpu_predictor", verbosity=1,
                            objective="binary:logistic", scale_pos_weight= 4)

# Create parameter grid
params = {"learning_rate": [0.2, 0.1, 0.01, 0.001],
               "gamma" : [10, 12, 14, 16],
               "max_depth": [2, 4, 7, 10, 13],
               "colsample_bytree": [ 0.8, 1.0, 1.2, 1.4],
               "subsample": [0.8, 0.85, 0.9, 0.95, 1, 1.1],
               "eta": [0.05, 0.1, .2, ],
               "reg_alpha": [1.5, 2, 2.5, 3],
               "reg_lambda": [0.5, 1, 1.5, 2],
               "min_child_weight": [1, 3, 5, 7],
               "n_estimators": [100, 250, 500]}

from sklearn.model_selection import RandomizedSearchCV

# Create RandomizedSearchCV Object
xgb_rscv = RandomizedSearchCV(xgb_clf, param_distributions=params, scoring='precision',
                              cv=10, verbose=3)

# Fit the model by running ten fits on ten 'folds', or 100 individual fits.
model_xgboost = xgb_rscv.fit(X_train, y_train)

However, during 50% of the 100 fits, I will get a score that looks like this:
[CV]  subsample=0.8, reg_lambda=2, reg_alpha=3, n_estimators=100, min_child_weight=3, max_depth=10, learning_rate=0.001, gamma=16, eta=0.1, colsample_bytree=1.4, **score=nan**, total=   0.1s

When this occurs, it occurs in sections of ten, so 10 straight fits will all generate a score of nan. The 50 nan scores don't always occur in the same order, but there are always 50 that don't get scored correctly.
Would anyone know how I can attempt to correct this and ensure that all 100 fits get scored? Is this happening because I'm using a hashed feature set?
Thanks!


texts: visualization idea on a panel data (roughly speaking)

texts:
I wanted to construct a heat map on the following data:

variable    bins    2000    2001    2004
    0       binA    2282447 2228013 2024397
    1       binB    264863  279788  355826
    2       binC    290533  304756  372788
    3       binD    211721  222480  257667
    4       binE    134966  142113  156282

Ideally I want (bins, bins) in a scatter plot or a heatmap. This is time series so to speak panel data (roughly saying). Any visualization idea on this?


texts: Predicting a numerical value based on past values and categorical attributes

texts:
I have training data consisting of a time series of numerical values (e.g. a user activity metric on a website over a period of 100 days).
I also have some categorical attributes of the user (demographics, type of browser, location, etc)

What would be a good approach to predict for a new user (say on day 50) the future value of the activity metric, either as a numeric value or as a class (low, medium, high)?

I could think of the following approaches:

A. Perform time series regression or classification separately for most (or all) combinations of the initial categorical features.

or

B. Extract 4 to 10 different numerical and/or categorical features from time series (e.g. high, low, average, weekend levels, weekend to weekday ratio, etc), then performing a random forrest classification or regression on the entire set of features.

or

C. Something else?

Thank you!


texts: Should output data scaling correspond to the activation function's output?

texts:
I am building an LSTM with keras which have an activation parameter in the layer. I have read that scaling on the output data should match the activation function's output values.

Ex: tanh activation outputs values between -1 and 1, therefore the output training (and testing) data should be scaled to values between -1 and 1. So if the activation function is asigmoid the output data should be scaled to values between 0 and 1.

Does this hold for all activation functions? If I use ReLu as activation in my layers what should the output data be rescaled to?


texts: What strategies and algorithms are suited for using the time wasted in collecting big data?

texts:
Most state of art algorithms right now is using/exploiting big data. My concern is what can you do to maximize reward while waiting for large amount of data that you think is appropriate.

For example, in predicting if a customer will churn (the reward is the revenue from all customers), the usual practice is to wait for data to accumulate. Is there a way to exploit the incoming information before exploiting the big data in full blown machine learning?

An easier example: you want to identify A over B and vice versa. The reward will be the accurate prediction which will be given by the user.

I know it is  reinforcement learning problem but how will you implement it?
I know that an on line training of ML algorithm will not work because you don't even know the algorithm that is suitable for the problem.


texts: rectangular markers in bubble plot (Python)

texts:
I would like to make a 'bubble' plot, but with rectangular markers. Is it possible to do this in Python?

On x-axis should be the day of the week (Mon, Tue, Wed,...), on y-axis - counts, how many times people came to a restaurant, the area of rectangles - the number of people came to a restaurant. A height of rectangles is always the same, only a width changes.

So, one rectangle on the plot shows how many people how many times came to a restaurant on a particular day of the week. As axis y may have values > 30, bubbles are not suitable and I consider that rectangles works better.

A sketch of what I would like to draw is below.

Is it possible to draw rectangular markers or is there another good way for visualizing the same concept?

UPDATE:
I've already tried box and violin plots.
The problem with boxplots is that in my case a height of a box is quite small and a tail is long. There are a lot of points indicating outliers. I would prefer to see some difference in outliers instead of a single point. With a bubble plot, it is possible to use some customized function for bubble size, e,g, logarithm.


texts: Loading collections of datasets - Python code examples

texts:
Sometimes you might want to check your ideas on multiple datasets.
There are several places with datasets collections.
Question: Please share some Python scripts how to download multiple datasets from these (or other)
datasets collection ?
Ideally one  should be able to : 1) get datasets list 2) select some desired by conditions 3) download those selected. But if you have something different please share anyway.
For "openml" database - I have a script - see my own answer.
But I do have for other collections : Kaggle, uci ...

Here some examples of datasets collections:
https://www.openml.org/
https://archive.ics.uci.edu/ml/index.php
https://ieee-dataport.org/datasets
Каggle contains lots of datasets, there are also specific collections:
graph collections see list here https://mathoverflow.net/a/359449/10446  ,
many biological data is here : https://www.ncbi.nlm.nih.gov/gds


texts:
I am trying to change continuous data points to categorical by using binning. I know two techniques, i) equal width bins ii) bins with equal number of elements.
My questions are:

Which type of binning is appropriate for which kind of problem?
I use pandas for my data analysis task and it has pd.cut method for arbitrary binning which I use for equal wdith bins and pd.qcut method for bins with equal number of elements. The second function always produces very complicated bin boundaries (like, [(-28.004,795.8976],(795.8976,900.342]]). Is there any way to "control" the bin boundaries so that they look more meaningful to non-technical persons?

Thanks in advance.


texts: Naive Bayes model - Missing output variable for specific sample(s)

texts:
With Naive Bayes model can we discard a training sample if the value of the sample's output variable is missing?
And would that not make any difference to the parameter learning of the Naive Bayes model?


texts: How to Connect Convolutional layer to Fully Connected layer in Pytorch while Implementing SRGAN

texts:
I was implementing the SRGAN in PyTorch but while implementing the discriminator I was confused about how to add a fully connected layer of 1024 units after the final convolutional layer
My input data shape:(1,3,256,256)
After passing this data through the conv layers I get a data shape: torch.Size([1, 512, 16, 16])
Code:
class Discriminator(nn.Module):

  def __init__(self):
    super(Discriminator,self).__init__()
    self.sm = nn.Sigmoid()
    self.net = nn.Sequential(
        nn.Conv2d(3,64,3,padding=1),
        nn.BatchNorm2d(64),
        nn.LeakyReLU(0.2),

        nn.Conv2d(64,64,3,2,padding=1),
        nn.BatchNorm2d(64),
        nn.LeakyReLU(0.2),

        nn.Conv2d(64,128,3,padding=1),
        nn.BatchNorm2d(128),
        nn.LeakyReLU(0.2),

        nn.Conv2d(128,128,3,2,padding=1),
        nn.BatchNorm2d(128),
        nn.LeakyReLU(0.2),

        nn.Conv2d(128,256,3,padding=1),
        nn.BatchNorm2d(256),
        nn.LeakyReLU(0.2),

        nn.Conv2d(256,256,3,2,padding=1),
        nn.BatchNorm2d(256),
        nn.LeakyReLU(0.2),

        nn.Conv2d(256,512,3,padding=1),
        nn.BatchNorm2d(512),
        nn.LeakyReLU(0.2),

        nn.Conv2d(512,512,3,2,padding=1),
        nn.BatchNorm2d(512),
        nn.LeakyReLU(0.2),

        nn.Linear(<ADD AN INPUT SHAPE HERE>,1024),
        nn.LeakyReLU(0.2),
        nn.Linear(1024,1)
    )

  def forward(self,x):
    x = self.sm(self.net(x))
    x = (x)
    return x


texts: Self Union All in Tableau Public for Sankey Diagram

texts:
I'm following this 2017 YouTube tutorial from which I hope to learn Sankey diagrams.
At 2:28, he instructs us to select Convert to Custom SQL, after having opened the spreadsheet in legacy mode - in order to do a Union All of SalesData (on itself).
Problems:

Legacy mode was not an option when opening
Convert to Custom SQL isn't in any of the options
All other Union options don't seem to allow for a Union All

Main Question:

How do I do a 'self' Union All on a table in Tableau Public?

Contextual Questions:

Is it that the Tableau UI has changed so much since 2017?
or, is this option not even available in Tableau Public?
or, is this just a bad direction for Sankey diagrams in the first-place?